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Abstract-  In  this  paper,  multi-input  computer  system  for 
image  acquisition  and  processing  is  presented.  This 
system  is  equipped  with  2-wavelength  thermal  channel 
and  any  4  other  one,  which  can  be  used  to  capture  visual, 
X-ray,  ultrasound  images,  etc.  Novel  image  processing 
tools  are  described,  mainly  for  geometrical  and  3D 
transformations. 
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I.  INTRODUCTION 


The  system  contains  analog  and  digital  part  as  shown  in 
Fig.  1.  In  the  analog  block,  two  16-bit  A/D  converters  are 
used  to  convert  thermal  and  visual  images.  CCD  flash  ADC 
is  equipped  with  4-input  analog  multiplexer.  It  makes 
possible  to  capture  visual  images  one-by-one,  with  the  delay 
of  one  frame  (20ms),  but  in  parallel  with  thermal  image.  In 
the  digital  part  1-2MB  buffer  VRAM  memory  is  used  to  store 
images  and  transfer  them  to  the  computer.  A  dual-port 
memory  is  suitable  for  fast  data  acquisition,  and  it  allows  to 
synchronize  easy  the  data  capturing  and  transferring  into  the 
operational  memory.  Entire  control  system  is  integrated  in 
high  density  FPGA  (Field  Programmable  Gate  Array), 
providing  high  flexibility  of  the  design  and  possibility  of 
system  reconfiguration  for  different  cameras.  Actually,  our 
system  is  prepared  for  thermal  cameras  with  both  the  analog 
and  the  digital  outputs. 


Sampling  clock  restoration  is  one  of  the  most  important 
circuit.  Sampling  rate  of  video  signal  (CCIRT-PAL)  at 
14.75MHz  ensures  the  square  pixel  and  the  proper  image  size 
on  the  screen.  Video  capturing  circuit  works  with  clock 
restoration  using  the  Digital  Phase  Lock  Loop  circuitry 
(DPLL).  This  causes  the  high  stability  of  the  restored 
sampling  clock  and  in  consequence  the  high  quality  of  the 
captured  image.  DPLL-based  restoration  circuit  is 
recommended  in  high-resolution  video  systems,  where  the 
slight  changes  in  the  phase  clock  can  disturb  the  captured 
image  smoothness  [3-4].  The  memory  buffer  implemented  as 
fast  dual-port  memories  (VRAM)  is  designed  as  a  FIFO 
(First-In-First-Out)  memory.  Input  data  coming  from  A/D 


converters  is  loaded  into  the  memory  through  Serial 
Register  (SAM).  The  corresponding  line  counter  for 
locating  the  memory  row,  where  the  pixels  are  placed  in,  is 
implemented  in  FPGA.  This  control  block  contains  also 
similar  counters  for  pixels  and  lines  to  read  data  from  the 
memory  and  transfer  them  to  the  computer.  The  main 
parameters  of  multichannel  system  are  presented  in  Table  I. 

TABLE! 


BASIC  SYSTEM  PARAMETERS 


Thermal  channel 

Visual  channel 

No.  of  channels 

1 

1-4 

ADC 

16-bit 

1 6-bit 

Resolution 

300x280  analog 
input/interlace 

256x256  digital/non¬ 
interlace 

640(720)/480 

interlace 

640(720)/240  non¬ 
interlace 

Memory  buffer 

1  MB 

2  MB 

DPLL  clock 
recovery 

yes 

yes 

Transfer  rate 

20MB/s  -  non  burst 
60MB/s  -  burst 

20MB/s  -  non  burst 
60MB/s  -  burst 

Real  time 
capturing 

yes 

yes  in  low  resolution 

Supported 

cameras 

Inframetrics®  Series 
600/700,  AGEMA® 
Series  600/700/800 
Hughes®  TVS4000, 
FLIR  SC  1000 

RSI 70,  CCIR-PAL 
Radiological, 
ultrasound,  visual 
cameras 

II.  3D  SCENE  RECONSTRUCTION 

3D  image  reconstruction  is  widely  used  in 
photogrammetry  applications  [3-5].  Up  to  now,  3D 
reconstruction  was  applied  effectively  for  video  images.  It 
seems  that  combining  video  and  thermal  systems  together 
with  3D  image  reconstruction  procedures,  provides  new 
possibilities  for  thermography  [3-4].  Localising  an  object  in 
3D  space  directly  allows  evaluating  a  distance  and 
orientation  of  thermal  specimen,  and  correct  object’s 
emissivity  and  ambient  transmission  for  infrared  radiation. 

There  are  well  known  methods  of  3-D  image 
reconstruction.  Methods  called  Shape  from  Shading  and 
Direct  Linear  Transformation  are  the  most  exploited  ones 
[1,4, 5, 6].  The  numerical  complexity  and  huge  amount  of 
information  to  be  processed  were  the  main  obstacle  for  the 
practical  implementation  of  3D  image  reconstruction.  New 
computer  technology  allows  realising  systems  for  such 
processing  today  [5]. 

Shape  from  Shading  method  assumes  the  illuminating  the 
object  from  certain  direction,  and  measuring  the  reflected 
radiation,  which  depends  on  the  reflection  angle  (Fig.  2-3). 
This  method  is  very  sensitive  to  the  color  and  non¬ 
homogeneity  of  the  surface,  as  well  as  the  illuminating 
radiation  characteristics. 
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Fig.  2.  “Shape  from  shading”  method  -  principle 


The  camera  setup  for  3D  image  reconstruction  is  shown 
Fig.4.  We  need  to  define  3D  coordinates  of  6  points  for  each 
image.  These  six  reference  are  used  for  camera  positioning. 
After  positioning  video  cameras,  any  extra  six  points  on  both 
video  and  thermal  pictures  should  be  marked.  These  extra  six 
points  may  be  the  same  or  different  as  reference  ones  used 
for  video  camera  positioning.  Program  automatically 
calculates  coordinates  of  this  extra  points  and  compute 
position  of  thermal  camera  without  any  given  coordinates  of 
marked  points  on  thermal  picture.  This  procedure  is  helpful 
when  it  is  difficult  to  locate  the  details  on  the  thermal  image. 
On  the  other  hand,  in  many  practical  cases  it  is  possible  to 
recognize  the  same  points  on  each  image,  including  the 
thermal  one,  without  providing  their  real  coordinates,  what  is 
also  the  3D  scene  reconstruction  algorithm.  In  the  final  stage 
of  the  algorithm,  groups  of  three  points  marked  by  the  user 
are  used  to  determine  the  chosen  planes  for  which,  angle  and 
distance  are  calculated,  and  used  later  for  emissivity  and 
temperature  correction. 


thermal  carrera 


Fig.  4.  Camera  setup  for  3D  scene  reconstruction 


Each  camera  gives  a  different  view  of  an  observed  object. 
Each  3D  point  visible  on  both  pictures  has  its  own  (different 
for  each  one)  2D  position.  Differences  between  this 
locations  on  pictures  obtained  from  different  cameras  are 
the  base  for  computing  the  3D  position.  The  picture  is  a 
result  of  transformation  of  3D  object  onto  a  flat  surface  of  a 
CCD  sensor  in  the  camera.  This  transformation  is  afinic 
what  makes  possible  to  present  the  dependence  between  the 
coordinates  of  point  in  3D  space  and  in  2D  picture. 

Typically,  two  or  more  video  cameras  take  pictures  of  the 
same  object,  so  we  can  write  set  of  such  equation  for  each 
point  and  camera,  and  solve  it  to  get  3D  coordinates.  Having 
more  than  2  cameras,  we  can  employ  the  least  square 
algorithm  to  improve  the  measurement  accuracy.  The  entire 
procedure  is  presented  in  Fig.  5. 


Fig.  5.  DLT  algorithm 


The  temperature  measured  by  the  thermal  camera  depends 
on  the  angles  between  the  normal  to  this  surface  and  the 
camera  axis.  The  measured  surfaced  may  be  marked  by 
three  points,  providing  a  surface’s  equation: 

AX  +  BY+CZ+D  =  0  (1) 

An  angle  77  between  camera’s  axis  and  the  normal  to  the 
surface  may  be  calculated  from  eqn.(l): 

[A  B  O']  Jrf-1  [0  0  l]r  (2) 


The  real  flux  emitted  by  a  piece  of  surface  may  be  finally 
corrected  using  Lambert’s  law,  as  below. 


<E  = 


cos  77 


(3) 


where:  d>0,  <f>  denote  the  flux  measured  by  the  camera, 
and  the  one  normal  to  the  surface,  respectively. 

Results  of  temperature  correction  are  presented  in  Fig.  6 
while  the  system  for  3D  reconstruction  is  in  Fig.  7. 


Fig.  6.  Directional  emissivity  correction 


Fig.  7.  System  for  directional  emissivity  correction 


II.  PERSPECTIVE  DISTORTION  CORRECTION 

In  many  practical  cases  an  object  is  too  close  to  the 
camera,  or  it  can  not  be  projected  non  perpendicular  onto  the 
detector  plane.  It  is  known  as  perspective  distortion.  In  order 
to  get  the  right  view  of  the  object,  plane-to-plane 
transformation  is  used  (Fig.  8).  This  method  needs  to  use  4 
reference  points,  as  it  has  8  transform  parameters  as  shown  in 
the  eqn.  (4). 

'  x  =  ul-x'  +  u2-y'i+u3 

u1  •  x\  +  u%  •  y\  + 1  (4) 

<  y  =  u4-xi+u5-yi+u6 
u7  •  x\  +  u%  •  y\  + 1 

where:  Ui  -  transform  coefficients;  X,Y  -  co-ordinates  after 
transformation  (real  co-ordinates);  x’,y’-co-ordinates  of  an 
original  pixel. 

The  better  results  can  be  obtained  using  more  reference 
points  and  applying  the  least-square  approximation  method. 
The  results  of  4-point  projection  are  presented  in  Fig. 9.  The 
original  image  is  a  projection  of  a  front  building  side  on  the 
plane  not  perpendicular  to  the  camera  optical  axis.  The 
windows  shown  there  are  not  of  the  rectangular  shape.  After 
transformation  of  the  front  building  plane,  on  the  plane 
normal  to  the  camera  axis,  the  windows  become  rectangular. 
The  entire  scene  keeps  the  real  aspect  ratio  significantly 
reducing  the  perspective  distortions.  After  such  correction  it 
is  quite  easy  to  stick  together  a  set  of  images  and  present  the 
whole  scene  in  the  orthogonal  projection  [1,5]. 


Fig.  9.  Example  of  perspective  distortion  correction 


Another  tool  makes  possible  to  overlay  visual  and  thermal 
images.  The  assumption  of  such  superimposing  is  that  the 
thermal  and  visual  images  are  projected  on  the  same  or 
parallel  planes.  We  can  easily  wrap,  shift  and  mix  two 
images.  It  simplifies  or  in  certain  cases  allows  locating 
precisely  the  hot  spots.  Two  images  (thermal  and  optical) 
allow  (by  using  common  cursor)  indicating  a  certain  point 
on  CCD  scene,  and  read  temperature  on  thermal  one. 

III.  SPECTRAL  EMISSIVITY 


Emissivity  was  introduced  to  characterize  the  optical 
properties  of  materials  in  sense  of  amount  of  enegry  emitted 
in  comparison  with  ideal  black  body.  Radiation  of  black 
body  is  described  by  emissive  power  eb(X,(p,0)  related  to 
power  emitted  from  unit  elemental  surface,  into  unit 
elemental  solid  angle  in  dX  spectral  range  (Fig.  10).  Unlike 
the  emission,  the  radiation  intensity  ib(X)  is  defined  on  the 
basis  of  projected  area  [1,2]  and  it  does  not  depend  on 
direction,  (Fig.  10).  Total  monochrome  emission  power 
passing  hemisphere  takes  a  form: 

In  2 

ebx=  h  W  [  f cos (<p)  sin ((p)dOd(p  =  mb  (X)  (5) 

(p=0d=0 

Total  radiation  intensity  can  be  evaluated  using  Planck 
law  as: 


Based  on  total,  monochrome  and  directional  quantities 
representing  the  amount  of  energy  emitted  we  can  define  the 
following  emissivities. 


Directional  monochromatic  emissivity 

_  Ci,0,<p  _  ^h,o,(p 


-'A,  9,<p 


eb,A,0,(P  eb,l,n  C0S(P 


Fig.  8.  Perspective  distortion  correction,  plane-to-plane  transformation 
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Monochromatic  normal  emissivity 
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Directional  total  emissivity 
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Hemispherical  total  emissivity 
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where  8^  is  the  hemispherical  spectral  emissivity  defined 
as: 

8,=r  fh  dody  (ii) 

A  Jo  Jo  x,(p,e 

Having  so  many  different  quantities  defining  emission 
properties  of  the  real  body  it  seems  to  be  quite  difficult  to 
evaluate  the  emissivity  in  practice  [2,6]. 

There  are  some  methods  for  emissivity  evaluation.  One  of 
the  first  method  of  emissivity  equalising  was  based  on 
covering  surface  by  thin  layer  with  high  and  known 
emissivity  factor.  Typically,  one  can  use  black  paint  with  the 
emissivity  factor  8=0.95-1  to  cover  measured  object.  This 
simple  method  has  basic  limit  of  its  application  -  covering 
even  by  very  thin  layer  changes  slightly  the  heat  transfer  in 
the  investigated  body  and  in  some  practical  cases  it  is  simple 
impossible,  e.g.  because  of  high  voltage. 

The  other  approach  to  emissivity  evaluation  is  based  on 
calorimetic  surveys.  So  far  we  know  the  object  temperature 
and  the  radiation  level  by  thermographic  measurement,  the 
simple  formulas  give  the  emissivity  factor  estimation  [6]. 

Using  the  system  described  above  we  can  evaluate  spectral 
emissivity  for  opaque  materials  by  measuring  the  its 
reflectivity,  as  expressed  below: 


n,h 


=  1  ~Pn,K 


(12) 


Heat  source 


infrared  camera 


Ein, 


d"%bj)  Einc 


amb 


object  under  measurement 
Fig.  1 1 .  Evaluation  of  emissivity  by  reflection 


In  order  to  evaluate  the  emissivity  by  reflection,  two 
measurements  must  be  performed.  First  one  is  to  calibrate  the 
system  by  measuring  the  heat  source  energy.  It  is  done  by 
reflection  from  the  infrared  mirror,  e.g.:  from  the  polished 
cooper.  The  energy  is  expressed  by  temperature  of  the  heat 
source  TCu.  Then,  the  radiation  energy  reflected  by  the 
investigated  sample  is  measured  Tm  (Fig.  11).  Including  the 
ambient  temperature,  and  assuming  the  very  high  reflectance 
of  the  mirror,  we  can  obtain  the  unknown  emissive  factor. 


ry4  rji  4 

£  =  l_  (13) 

ry r4  ry  4  7 

1  Cu  1  amb 

Some  thermal  cameras  are  equipped  with  broadband  (BB) 
detectors,  such  as  microbolometers  or  BB  photon  ones.  By 
using  build  in  the  camera  band  pass  filters  differentiating 
short- wavelength  (2-5|im)  with  long- wavelength  (8-12  jam), 
it  is  possible  to  evaluate  the  emmisivity  for  both  spectral 
ranges.  Having  the  emissivity  maps  for  investigated 
surfaces,  it  is  easy  to  correct  the  temperature  measured  the 
thermographic  camera  [6]. 

IV.  CONCLUSION 

The  system  for  thermal  and  visual  image  processing  is 
presented  in  this  work.  This  system  can  be  used  for 
capturing  images  in  real-time,  frame-by-frame,  or  storing 
the  sequence  with  defined  interval  between  the  following 
images.  It  is  actually  used  for  advanced  thermal  and  visual 
images  processing,  e.g.  3-D  photogrametry  reconstruction, 
noise  cancelling,  image  superimposing,  etc.,  and  improves 
the  thermal  camera  measuring  parameters. 

The  methods  presented  in  this  paper  give  the  ability  to 
reconstruct  3D  coordinates  of  measured  object,  including 
the  positions  of  all  cameras  used  for  the  measurements.  It 
allows  computing  both  the  angles  between  camera’s  axes 
and  normal  to  the  measured  surface  and  the  distance 
between  the  camera  and  the  object.  This  is  necessary  for  the 
correction  of  emissivity,  and  in  consequence  the 
temperature  measured  by  camera.  Although  the  methods 
presented  in  this  paper  are  based  on  well-known  algorithms, 
a  few  new  improvements  were  added  to  include  thermal 
system  for  photogrammetric  application. 
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